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ABSTRACT 

The Workforce Investment Act 1998 (WIA) made major changes in 
programs for youth from economically disadvantaged families. It mandated 
program performance indicators. A study examined the new performance 
indicators and determined whether they were adequate or should be 
supplemented with involvement of stakeholders concerned about youth programs . 
In 9 of 12 Ohio School-to-Work (STW) regions, 108 stakeholders took part in 
10 focus groups and discussed what indicators, beyond those mandated, should 
be used to evaluate youth programs. Content analysis of discussions yielded 
four potential performance indicators that would supplement, if adopted, 
those that legislation mandated. The indicators are that program completers 
can articulate career goals and plans for achieving them; demonstrate the 
ability to work well with others; manage their personal lives effectively; 
and demonstrate a good work ethic. Only 24 of 108 participants completed 
rankings of the ' indicators . Survey questionnaires circulated to 47 providers 
asked whether they collected data on the indicators and, if not, their 
preference for how it should be collected; 9 providers responded. The low 
response rates indicated data use for program improvement is not a high 
priority for providers. To increase awareness and understanding of potential 
contributions of performance data, an introductory guide to data collection 
and use was prepared and circulated to providers. (The guide is appended.) 
(YLB) 
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Foreword 



The Center on Education and Training for Employment is pleased to present 
this report on potential indicators for youth programs funded by the Workforce 
Investment Act of 1998. This report was produced by one component of a 
broad, multi-university initiative, the State University Education Deans School- 
to-Work Systems Integration Co^ition that was funded by the Ohio 
Department of Education. Ms. Kristen Cox Howard was the project officer for 
the Coalition. 

Prior to her retirement from The Ohio State University, Ms. Sandra Pritz 
provided coordination for the Coalition and co-directed the project that 
produced this report. After her retirement, the co-project director. Dr. Morgan 
Lewis, and Ms. Susan Resch completed the project activities and prepared this 
report. The activities of the Systems Integration Coalition were guided by an 
advisory board with representatives of all major groups concerning with 
transition from education to employment. Three members of this board served 
as facilitators for focus groups conducted for this project: David Booth, Qetler 
Jensrud, and Otto Meyer. 

The staff has asked me express their appreciation to all who contributed to the 
project, especially the 108 individuals who took part in the focus groups. 

These participants were from across Ohio and all are involved in programs for 
youth. School-to-Work and Tech Prep coordinators and staff members of Full 
Service Centers identified potential participants and assisted in arranging the 
sessions. The discussions in the focus groups produced the basic data from 
which the indicators proposed in this report were developed. 

I also wish to recognize the contributions to this report of these program 
providers: Ms. Joann Davis, Director Village to Child Program, Ohio Dominican 
College, Columbus, Ohio; Ms. Sally Murphy, Head of Youth Outreach, and Ms. 
Sarah Narendorf, Head of Transitional Living Program, Huckleberry House, 
Columbus, Ohio: and Ms. Kathryn Meyers, Direetor Marketing and 
Development, Salesian Boys’ and Girls’ Club of Columbus. 

On behalf of all of us at Ohio State, I express our sincere appreciation to those 
whose concern and dedication to young people motivated them to cooperate in 
the research presented in this report. It is our hope that the results will be 
helpful to all providers of youth services. 

W. Michael Sherman 
Interim Executive Director 
Center on Education and Training 
for Employment 
College of Education 
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Executive Summary 



The Workforce Investment Act (WIA) of 1998, which became effective July 1, 
2000, made major changes in programs directed to youth from economically 
disadvantaged families. Under the prior legislation, youth programs primarily 
provided summer employment opportunities, often supplemented with an 
educationad component. Under WIA these programs are required to offer a 
comprehensive array of services year around, continuously from the time 
young people enroll until they complete or withdraw from their programs. 
Toge^er with these new programming requirements, indicators of program 
performance are also mandated. 

In October 1999, in anticipation of these changes, the Ohio School-to-Work 
Office asked, the Center on Education and Training for Employment to 
examine the new performance indicators and determine if they were adequate 
or should be supplemented. The examination was conducted with the 
involvement of a broad array of stakeholders concerned about youth programs. 
This involvement took the form of focus groups, surveys, and the development 
of a guide to the use of data for program improvement. 

Focus groups were held in nine of the 12 state School-to-Work (STW) regions. 
The groups were convened with the assistance of the STW Coordinators, the 
Tech Prep Coordinators of consortia most aligned with the regions, and staff of 
the Full-Service Centers in the regions. A total of 108 stakeholders took part in 
10 focus groups. (One regions had two groups because of the number who 
responded to the invitation to participate.) 

The Nominad Group Technique was used in these focus groups to discuss this 
question: 

What additionad indicators, beyond those mandated by the WIA, should 
be used to evaduate youth programs? 

Content anadysis of the discussion in these groups yielded the following four 
potentiad indicators of program performance. If these indicators were adopted, 
they would supplement those mandated by the legislation: 

1. Program completers can articulate career goals and plans for achieving 
them that are based on a realistic assessment of their interest, abilities, 
and the resources available to them. 

2. Program completers demonstrate the ability to work well with others. 
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3. Program completers manage their personal lives effectively, including 
budgeting, time management and accessing resources needed to 
overcome barriers to education and employment. 

4. Program completers demonstrate a good work ethic including 
punctuality, regular attendance, and doing what needs to be done. 

After these four indicators had been drafted, they were circulated to all who 
had participated in the groups, and they were asked to rank all that they 
thought should be used for program ev^uation from 1 , their highest rank, to 4, 
their lowest. Only 24 of the 108 participants completed the rankings, but 23 of 
the 24 ranked all four. In other words, all but one of the focus group 
participants who returned rankings thought that all four indicators should be 
used for program evaluation. The work ethic indicator, number 4 in the list 
above, was the highest ranking, receiving a mean of 1.54. This was over a full 
point lower than the next lowest mean of 2.65 for the indicator on career 
planning. 

The next step in the project was a survey of providers of youth program in the 
Central Ohio area. Questionnaires listing the four indicators developed from 
the focus groups were circulated to 47 providers. The questionnaires asked if 
the providers currently collected data on these indicators, and if they did not 
their preference for how it should be collected. The response to the survey was 
very low. Only nine providers returned the questionnaire, and of these, only 
four reported they currently collect data related to any of the indicators. 
AttemptS’were made to conduct interviews with the four providers who collect 
the relevant data, but only two agreed to the interviews. 

The low response rate to the survey and the two interviews that were 
conducted led to the conclusion that, at present, use of data for program 
improvement is not a high priority for providers of youth services. To increase 
awareness and understanding of the potential contributions of performance 
data, an introductory guide to data collection and use was prepared and 
circulated to the same providers who had been surveyed. 

The providers were asked if they would be willing to review the guide and 
provide suggestions for its improvement. One provider responded to this 
request and reported that she found the guide to be very helpful. She 
suggested topics in the guide that should receive additional emphasis, and 
these suggestions were incorporated. The guide is presented in the Appendix 
of the report. 
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INTRODUCTION 



For the past several years, the College of Education of The Ohio State 
University has provided coordination for the School-to-Work Systems 
Integration Coalition. This coalition was a collaborative effort of the State 
University Education Deans, the Board of Regents, and the Department of 
Education. It was developed to encourage broad participation from all state 
teacher education institutions in school-to-work initiatives. 

In October 1999, the Ohio School-to-Work (STW) Office asked Ohio State, as 
part of its coalition activities, to examine the performance indicators that would 
be applied to youth programs funded under the Workforce Investment Act 
(WlA) of 1998. This act, which replaced the Job Training Partnership Act 
(JTPA), was passed in August 1998. To allow time for transition to tiie new 
provisions, it did not become effective until July 1, 2000. 

The examination of the performance indicators was conducted through focus 
groups and surveys of individuals concerned about youth programs. Focus 
groups were conducted across the state with individuals who were convened 
through the assistance of STW Coordinators, Tech Prep Consortium 
Coordinators, and staff from the Full-Service Centers of the Office of Career- 
Technical and Adult Education, Ohio Department of Education. A follow-up 
survey was conducted with participants from these groups. Another survey 
was conducted with providers of youth programs in Central Ohio and attempts 
were made to conduct follow-up interviews with providers. Finally, a guide to 
the use of data to improve programs was prepared and circulated to youth 
providers in Central Ohio for comment. 

This report describes each of these activities and the results they produced. It 
is organized in the order in which the activities were carried out: 

■ Statewide focus groups 

■ Survey of focus group participants 

■ Survey and interviews with Central Ohio providers of youth programs 

■ Guide to use of data for program improvement 

Before discussing these activities, a brief overview of the Workforce Investment 
Act (WLA) is presented. 



Workforce Investment Act 

WlA made major changes in the requirements for youth programs. Under 
JTPA, these programs consisted primarily of summer employment for young 
people from low-income families. Classroom activities were sometimes 
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provided, but these were secondary to employment. Under WIA, youth 
programs still focus on disadvantaged youth, but those who are enrolled must 
participate during the entire year and have access to the following program 
elements: 

• Tutoring, study skills and instruction 

• Alternative secondary school services 

• Summer emplo)mient 

• Work experience 

• Occupational skill training 

• Leadership development 

• Supportive services 

• Adult mentoring 

• Follow-up services 

• Guidance and counseling 

WIA does not require that all enrollees receive all of these services, but 
programs must have ways of making all of them available to those enrollees 
who need them. An individualized plan must be developed for each enrollee 
that sets at least three measurable goals. The program elements from this list 
of 10 that will contribute to the attainment of an enrollee’s goals are to be 
included in each plan. 

Young people in the age range of 14 to 21 can be enrolled. Those who are 19 to 
21 years of age can be served either as youth or adults. If they are in youth 
programs, they can receive the services listed above, but they are not eligible 
for individualized training accounts. Young people 14 through 18 years of age 
are eligible only for youth programs. WIA specifies three broad categories of 
core indicators by which youth programs are to be evaluated: 

1. Attainment of basic sldlls as appropriate, work readiness and occupational 
skills 

2. Attainment of secondaiy school diplomas and their recognized equivalents 

3. Placement and retention in post secondary education or advanced training, 
or placement and retention in military service, employment or qualified 
apprenticeship 

State and local workforce investment boards are established under WIA to 
provide policy guidance to the implementation of the act. These boards have 
the option of adopting additional indicators for use in evaluating the programs 
they fund. The School-to-Work Office asked Ohio State to conduct a^study to 
determine if those involved in youth programs across the state saw a need for 
additional indicators, and if they did, what those indicators should be. 
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FOCUS GROUPS 



The focus groups for this project were conducted during the last four months of 
transition from JTPA to WIA, March through June of 2000. In Ohio, this 
transition was taking place while the Ohio Bureau of Employment Services was 
being merged with the Ohio Department of Human Services into a new 
department, the Ohio Department of Job and Family Services. Many of the 
local agencies that had responsibility for administration of JTPA were 
eliminated and their functions assumed by the county Departments of Job and 
Family Services. 

Because of the state-level merger and the reorganizations at the local level, the 
Youth Councils that are required by the WIA and charged to advise state and 
local workforce investment boards had not been appointed when the focus 
groups were conducted. The question that arose, therefore, was whom to call 
upon as representatives of youth programs. A decision was made to request 
assistance in convening focus groups from three sources: the School-to-Work 
(STW) Coordinators in the 12 regions, the Tech Prep Coordinators whose 
consortiums were most congruent with these regions, and the Full-Service 
Centers in the regions. 

Requests for cooperation were sent to all 12 regions and focus groups were 
conducted in 9 of the 12. Because of the number who responded to the 
invitation to take part in the focus groups, two groups were conducted in 
Dayton. The number that responded in Lima was also large enough for two 
groups, but the facility in which the group met could only accommodate one 
large group. Table 1 indicates the dates, locations, and number participating 
in each of the focus groups and Table 2 presents the characteristics of these 
participants. 



Table 1. Focus Groups Participation by Date and Location 



Date 


Region 


Location 


Participants 


March 14 


8 


Cleveland 


7 


March 2 1 


7 


Piketon 


12 


March 28 


2 


Toledo 


11 


April 4 


3 


Lima 


17 


April 4 


4A 


Dayton 


13 


April 4 


4B 


Dayton 


9 


April 5 


6 


Mansfield 


12 


April 12 


1 


Columbus 


12 


June 6 


10 


New Philadelphia 


8 


June 22 


11 


Athens 


7 



Note: All dates are in the year 2000. 
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Table 2. Demographic Characteristics of Focus Group Participants 



Characteristic 


Percentage 


Race/ Ethnicity 




White 


79 


African-American 


9 


Other 


3 


No answer 


9 


Gender 




Female 


44 


Male 


47 


No answer 


9 


Age 


8 


35 or younger 


36 to 45 


19 


45 to 55 


46 


56 or older 


17 


No answer 


9 


Base Number 


108 



The focus groups were conducted using the Nominal Group Technique (Delbecq 
and others 1975). This is a structured method of generating and prioritizing 
options for addressing a specified issue. The issue that these groups were 
asked to consider was the following: 

What additional indicators, beyond those mandated by the WIA, should 

be used to evaluate youth programs? 

Prior to the discussion, the information sheet on youth indicators reproduced 
as Exhibit 1 was distributed to the participants. After a brief explanation of the 
method, the participants were asked to take a few quiet minutes to think of 
additional indicators. Each participant was then asked to suggest an indicator 
for listing on a flip chart. The listing was done in round-robin fashion until no 
additional indicators were suggested. One of the project staff either facilitated 
the groups or acted as recorder. 

After all indicators were listed, the group had the opportunity to review and 
cluster similar indicators. In most cases, this process resulted in a large 
number of indicators, 30 to 40, being reduced to a smaller number. This 
clustering often yielded a somewhat disjointed aggregation of similar , but far 
from identical concepts. The participants were then asked to select from those 
remaining indicators, some of which were clusters, the five they most preferred 
and to rank those five with number 1 being their most preferred. 



O 

ERIC 



4 



11 



Exhibit 1. Summary of Workforce Investment Act 
Performance Indicators for Youth Programs 

Core Indicators of Performance: (WIA Sec. 136(b)(2)(A)) 

For providers serving youth ages 19-21: 

1. Eligible youth who enter postsecondaiy education, advanced training, or 
unsubsidized employment. 

2. Eligible youth who enter an activity listed in indicator 1 who received a 
credential by the end of the third quarter after exit. 

For providers serving youth ages 14-18: 

1. Attainment of basic skills as appropriate, work readiness and occupational 
skills. 

2. Attainment of secondary school diplomas and their recognized equivalents. 

3. Placement and retention in post secondary education or advanced training, or 
placement and retention in military service, employment or qualified 
apprenticeship. 

Customer Service Indicators: 

The customer satisfaction indicator of performance shall consist of customer 
satisfaction of employers and participants with services received from the workforce 
investment activities authorized. Customer (employer and participants) satisfaction 
may be measured by surveys conducted at the conclusion of participation in the 
workforce investment activities. The American Customer Satisfaction Index (ratings 
on 3 scales from 1 to 10) has been adopted for the core measure. 

Additional Indicators: 

The State may identify in the State plan additional indicators for workforce investment 
activities. Local boards may adopt indicators beyond those required by the state. 

Target Population Groups: 

Youth must be ages 14-21, low income, and meet at least one of these six specific 
barriers to employment: 

1. Deficient in basic literacy skills. 

2. A school dropout. 

3. Homeless, a runaway, or a foster child. 

4. Pregnant or a parent. 

5. An offender. 

6. An individual who requires additional assistance to complete an educational 
program, or to secure and hold employment. 

Five percent may be non-low income if they have specified barriers to school 
completion or employment. At least 30 percent of the funds must be spent on out- of- 
school youth. 

Youth Councils: 

Youth Councils will be established as a subgroup of the local board to develop parts of 
the local plan relating to youth, recommend providers of youth services, and 
coordinate youth programs and initiatives. 
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Analysis of Focus Group Results 

After all 10 focus groups had been conducted, it was necessary to compare the 
indicators that had been suggested and ranked by each. The method chosen 
for this comparison was content analysis using Ethnograph (Seidel, 1998), a 
program for personal computers that facilitates content analysis of text. The 
codes used for this analysis were developed from a review of the high-ranking 
indicators that emerged from each focus group. The highest rankings were 
those selected and ranked by a majority of each group. This criterion 
eliminated those indicators that had been selected by less than half of the 
group. Exhibit 2 presents the codes that were used for analyzing the high- 
ranking indicators. 

Exhibit 2. Codes Used for Analysis of Potential Indicators 



ACHIEVE: Achievement of a specified goal, may be nontraditional. 

BASIC SKILLS: Basic academic, communication skills. 

CAREER: Ability to plan and carry out career plans. 

INTERPERSONAL: Ability to work well with others, team work. 

OCCUPATION SPEJCIFIC: Occupationally specific skills, e.g., computer skills, defined 
competencies. 

PERSONAL: Personal management skills, time management, budgeting, problem 
solving, anger management, conflict resolution 

RESOURCES: Ability to access resources to deal with personal problems 

RESPONSIBILITY: Work ethic, sense of responsibility, attendance, punctuality 

The coding process sought to identify the underlying concepts in the indicators 
from each group, regardless of how these concepts were worded. It was noted 
above that as part of the focus group process, participants were able to review 
and cluster separate indicators. Here is an example of an indicator that 
emerged from one of the focus groups that clustered several separate indicators 
into one: 

Attendance, understanding of it and dependability. Good understanding 
of importance of developing a quality resume, job finding tools. Life skills, 
budgeting. Learn how to work as a team, people skills. The ability to 
understand and take direction. Time management schedule skills, living 
by a schedule. Anger management skills. Principles of honesty and 
integrity. Good grooming, proper attire or uniform, showing up for work. 
Realization that there is no "free lunch,” understanding limitations and 
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expectations. Dedication and work ethic. Talking through problems at 
work with supervisors. Managing money, finances. 

This indicator was selected by 10 of the 12 participants in this group and 
received the highest average ranking, 1.7, on a sc^e of 1 to 5. The indicator 
received such a strong endorsement because it included several references to 
desirable outcomes of youth programs. Here is how this indicator was coded: 

The coding 5 delded five separate references that were coded responsibility: 

• Attendance, understanding of it and dependability 

• Good grooming, proper attire or uniform, showing up for work. 

• Principles of honesty and integrity 

• Realization that there is no "free lunch", 

• Dedication and work ethic. 

Four that were coded personal: 

• Life skills, budgeting 

• Time management schedule skills, living by a schedule. 

• Anger management skills. 

• Managing money, finances. 

Three that were coded as interpersonal: 

• Learn how to work as a team, people skills. 

• The ability to understand and take direction 

• Talking through problems at work with supervisors. 

Two that were coded career. 

• Good understanding of importance of developing a quality resume, job 
finding tools 

• Understanding limitations and expectations. 

This indicator is a rather extreme example of clustering. Most indicators did 
not have so many separate concepts included. 

For nine of the ten focus groups, two analysts coded the indicators 
independently, and the codes they assigned were compared. The indicators 
from one group were used for training purposes to develop consistency in the 
manner in which the two analysts assigned the codes. For the nine that were 
coded independently, there was 85 percent agreement in the codes they 
assigned. This is equivalent to a reliability coefficient of r= .92. The codes on 
which there was initial disagreement were discussed and agreement reached. 
Exhibit 3 presents the wording from each of the indicators that was grouped by 
the coding system presented in Exhibit 2. The group producing the wording is 
identified by region (R). The two groups conducted in Region 4 are labeled A 
and B. 
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Exhibit 3. Coding of Basic Concepts Reflected in Indicators 
by Regions in Which the Focus Groups Were Conducted 



ACHIEVE: Achievement of a specified goal, may be nontraditional. 

Rl: Increase in self-esteem. 

R2 High school diploma or equivalent for ages 19-21. 

R3; More non-traditional alternatives, educational opportunities, realistic approach 
to achieve graduation or equivalent. 

- Meaningful recognition of steps of achievement; how to achieve to next level of 
proficiency; celebration of accomplishment. 

R4A Experience successful work-based learning. 

- Driver's license. 

RIO:' Obtain a valid driver's license. 

Rl 1 Stay with 3 listed, [indicators in the WIA] measure academics by state 
proficiency tests. 

- Continuation in a program that will lead to a high school diploma or 
equivalent after age 18. 

- Completion of structured program designed to increase feelings of 
accomplishment, self-esteem, success. 



BASIC SKILLS: Basic academic, communication skills 

Rl: Include strong math, science and reading skills. 

R2: Reading, writing, mathematical skills at specified grade level. 

R4A: High School completion standards, high school diploma, math, English classes, 
GED, 9th grade proficiency exams passed. 

R4B: Ability to read and write. 

- Socially acceptable communication skills in a business environment. 

R6: Basic computer literacy/keyboard skills, basic communication skills; 

communicate with others, core skills in reading, writing, math. 

R7: Improved proficiency, reading, writing, math, comprehension, science, improved 

writing and learning comprehension skills. 

R8: Demonstrate ability to read, comprehend, and apply knowledge. 

Respond inteUigently to questions. 

RIO Communication skills: Able to communicate professionally both written and 
orally. Have telephone etiquette. Ability to ask relevant question. Listen well. 

- Basic Skills: Has ability to read and write basic English. Can perform basic 
math skills. Shows citizenship skills. Ability to follow directions/ orders. 
Understands technology fundamentals. Has basic keyboarding skills. 

Develops basic literacy skills. Takes advantage of lifelong learning. Understands 
free enterprise system. 

Rl 1: Stay with 3 listed, [indicators in the WIA] measure academics by state 

proficiency tests. 



Exhibit 3. Coding of Basic Concept, Continued 



CAREER: Ability to plan and carry out career plans. 

Rl: Good understanding of importance of developing a quality resume, job finding 

tools. 

R4A: Increased knowledge of career awareness and employer expectations. Ability to 
set goals. 

No "free lunch", understanding limitations and expectations. 

R4B: Career competency attainment, career awareness, job-ready. Realistic 
awareness and expectations of career field. Student doing realistic self- 
assessment; able to match with real jobs. 

- Have a career plan, first step and long term. See at a first step of a career 
path with future growth and learning. Get away from the concept of working for 
money's sake, but outcomes of staying with program and training in the long 
term. 

- Career exploration. 

R7: Clear career goals and plan to achieve increased awareness of one's personality, 

learning styles, interpersonal interaction. 

R8: Completed a jobs workshop. Fill out a job application. Basic writing skills (e.g., 

can write cover letter, thank you note). Understanding of job opportunities, 
market, related jobs. Demonstrate proper job interview techniques. 

Appropriate telephone skills, call those they don't know. Develop a resume. 
Occupational skills to advance. Appropriate dress. Awareness of SCANS 
competencies. Look up/research something. 

- Ability/ willingness for lifelong learning. Ability to network, 

- Identify a career focus and an educational plan to reach the goal. 

RIO: Articulate personal goals . 

- Exhibit interviewing skills. Can indicate their skill level correctly to 
employers. Be familiar with area of employment being sought. 



INTERPERSONAL: Ability to work well with others, team work. 

Rl: Learn how to work as a team, people skills. The ability to understand and take 

direction. 

- Talking through problems at work with supervisors. 

- Not being afraid to ask questions. Initiative, willingness to learn and ask 
questions. 

R3: Ability to get along with others. 

R4A: Skills in anger management, conflict resolution and peace keeping. Able to 
function as a part of a group/ team. 

R4B: Interpersonal skills, able to work in teams, peer expectations, Ability to work in 
groups, not just peer groups, but adults, Awareness of peer employee 
expectations. Ability to work successfully in a diverse work force. 
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Exhibit 3. Coding of Basic Concept, Continued 



R6: Communicate with others, team skills. 

- Social skills; get along with others, Cultural/ gender diversity awareness and 
sensitivity. 

- Fellowship and leadership. 

R8: Ability to accept constructive criticism. 

- Ability to work under authority. Ability to follow directions, written or verbal. 
RIO: Interpersonal skills, demonstrates the ability to get along with others. Able to 

accept others who are different than themselves. Clearly exhibit conflict 
management skills. 

- Accept criticism, give and receive feedback. 

R1 1 : Employability work readiness. Working with coworkers, working with 
customers. Social skills-getting along with others/ team player. 



OCCUPATION SPECIFIC: Occupationally specific skills, e.g., computer skills, 
defined competencies. 

Rl: Basic computer and drafting skills. 

R4A: 1 1th- 12th grade technology skill training "marketplace dictates" Marketing 

skills. Evidence of a set of competencies in the occupational area beyond 
minimum wage. 

- Suitable industry certification, readiness. 

RIO: Obtained specific vocational skills and can demonstrate those specific skills to 
an employer. 



PERSONAL: Personal management skills, time management, budgeting, problem 
solving, anger management, conflict resolution 

Rl: Life skills, budgeting. 

- Time management schedule skills, living by a schedule. Anger management 
skills. Principles of honesty and integrity. 

- Managing money, finances. 

- Problem solving. 

- Increase resiliency factors. 

R3: Independent living skills like budgeting and time management. 

R4A: Cleeinliness, personal habits. 

- Drug free. Not becoming a parent. Avoidance of involvement with juvenile 
justice system. 

R4B: Time management skills. 

- Staying with the effort. 
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Exhibit 3. Coding of Basic Concept, Continued 



R4B: Awareness/ demonstration of employability skills, employer expectations. 

- Awareness of what employer expects of a new employee. Be trustworthy. 

R6: Work ethic, punctuality, grooming, responsibility of a personal nature, 

attendance. Follows directions/ rules. Expectation that they can work with all 
others. Ability to work on their own without constant supervision. Dedication 
as an employee, commitment to the goals/ objectives. 

- Pride in quality work. Attitude that enables following through on tasks and 
working for others. 

R7: Ethical behavior, including work ethic/ character. Responsible/ reliability as 

worker. 

R8: Exhibit work ethic. 

RIO: Exhibits work ethic. 

R1 1: Shows up on time. Dresses appropriately, appropriate language. 

- Review of youth work performance by employers. 

- Stay with 3 listed [indicators in the WLA], measure academics by state 
proficiency tests. 



It was noted in the Introduction that the WLA specifies three core indicators for 
youth programs serving the age range 14 to 18: 

1 . Attainment of basic skills as appropriate, work readiness and occupational 
skills. 

2. Attainment of secondary school diplomas and their recognized equivalents. 

3. Placement and retention in post secondary education or advanced training, 
or placement and retention in military service, empIo 5 anent or qualified 
apprenticeship . 

The indicators that emerged from the focus groups can, for the most part, be 
classified within these three. If the work readiness skills in the first indicator 
are defined broadly, the focus groups primarily yielded further definition and 
examples of these skills. All of the skills classified as career, interpersonal, 
personal, resources, and responsibility contribute to work readiness. To provide 
further specificity as to just what work readiness involves, four potential 
additional indicators could be added to the three mandated in the legislation. 
The indicators listed in Exhibit 4 can be considered as providing further 
specificity to the core indicators. 

The other basic concepts that emerged from the focus groups directly reflect 
the three core indicators. The achieve concept is inherent in the goal setting 
that is part of the development of the individualized plan for each enrollee. 
Programs are to be evaluated on the extent to which all of their enrollees 
achieve the goals set forth in their individualized plans. Basic and occupational 
skills are specifically addressed in the first core indicator. 
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E^ibit 3. Coding of Basic Concept, Continued 

R6: Life management skills, balance of work with other aspects/roles in life. 

Attention to balance of physical, mental, spiritual aspects of life, a belief system 
of a spiritual nature. 

- Problem solving skills. 

— Management of personal life so that job impact is positive; can work through 
problems. 

- Ability to accept criticism, build on it. Self-esteem, self-assessment and 
evaluation skills. 

- Experiences in management of selves. 

- Adaptability, expectation of multiple jobs/roles and ability to cope. 

R7: Practice healthy life style (wellness). Reduction in alcohol/drug usage. 

- Demonstrated respect for other workers, types of work. 

R8: Demonstrate life skills. Respond intelligently to questions. Creative. Money 

management skills. 

- Problems solving (without direction). 

RIO; Exhibit time management skills. Cleanliness. Able to problem solve in all 
environments. 

- Successful life skills. Be drug free. Developed effective parenting skills. Be 
economically literate. Be in good standing with the justice system. 



RESOURCES: Ability to access resources to deal with personal problems 

R4A; Nurturing relationships and mentoring. 

R6: Resource awareness. Review barriers to employment-related education 

(transportation, housing, day care, clothing). Identify barriers to success and 
resources for addressing this. Access to community resources. 

RIO: Have child care established. Access clean, affordable, reliable housing. 
Establish their own personal support system and network. 

- Accepts referral to special needs providers for Work Adjustment and Sheltered 
Employment. 



RESPONSIBILITY: Work ethic, sense of responsibility, attendance, punctuality 

Rl: Attendance, understanding of it and dependability. 

- Proper attire or uniform, showing up for work. 

- Dedication and work ethic. 

- Do they do things they see need to be done? 

R3; Need to understand work ethics, reliability, attitude. 

R4A; Employability skills. Reduced absenteeism. 

- Punctuality. 
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Exhibit 4. Potential Additional Indicators 
of Youth Program Performance 

Program completers can articulate career goals and plans for achieving them that 
are based on a reahstic assessment of their interest, abilities, and the resources 
available to them. 

Program completers demonstrate the ability to work well with others. 

Program completers manage their personal lives effectively, including budgeting, 
time management and accessing resources needed to overcome barriers to 
education and employment. 

Program completers demonstrate a good work ethic including punctuality, regular 
attendance, and doing what needs to be done. 
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SURVEY OF FOCUS GROUP PARTICIPANTS 



The next step in the process was to give all those who participated in the 10 
focus groups a chance to review the four proposed indicators that were 
developed on the basis of their discussions. The indicators were sent to all the 
participants and they were asked to rank from 1 to 4 those that they believe 
should be recommended for use in evaluation of youth programs funded by 
WlA. A previous draft of this section of the present report was sent to inform 
the participants of the process used to develop the four potential indicators. 
The form used in the survey is shown as Exhibit 5. 

Exhibit 5. Form Used by Focus Group Participants 
to Rank Potential Indicators 



Directions: Listed below are the performance indicators discussed in the report sent 
with this form. Please rank those indicators that you think should be used for youth 
program evaluation. Rank them with 1 being your most preferred, 2 your next most, 
and so on. You do not have to rank all the indicators listed. Rank only those that you 
think should be recommended for use in evaluating youth programs funded by the 
Workforce Investment Act. 



Rank 

Program completers can articulate career goals and plans for 

achieving them that are based on a realistic assessment of their 

interest, abihties, and the resources available to them. 

Program completers demonstrate the abihty to work well with 

others. 

Program completers manage their personal lives effectively, 

including budgeting, time management and accessing resources 

needed to overcome barriers to education and employment. 

a 

Program completers demonstrate a good work ethic including 

punctuality, regular attendance, and doing what needs to be done. 



As shown above, the form asked the participants to rank only those indicators 
that thought should be recommended for use in evaluating youth programs 
funded by the WlA. All but one of the returned forms ranked all four, 
suggesting a general acceptance of the appropriateness of all the indicators. 
The response rate to the survey, however, was not good. A total of 108 had 
participated in the focus groups, but only 24 returned rankings. Repeated 
follow-up contacts were made by e-mail, telephone, and fax, but few additional 
responses were received. Part of the reason is that the survey was conducted 
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during the summer months of 2000. Many of the participants in the focus 
groups were educators and they were not at their contact numbers during the 
summer. Those that did respond were fairly evenly distributed across the nine 
regions where the focus groups were conducted. 

Table 3 presents the average rankings of the four indicators. The respondents 
had been asked to assign a rank of 1 to their most preferred indicator, 2 to 
their second choice and so on. Consequently, the lower the mean the higher 
the indicator was ranked. The work ethic indicator was clearly the most 
preferred. Its average ranking is more than a full point below the mean for the 
indicator concerning career planning. This difference is statistically 
significantly at the .01 probability level, but the other three means are not 
significantly different from one another. 

Table 3 

Mean Rankings of Potential Additional Indicators 
Of Youth Program Performance 



Potential Indicator 


Mean 


Standard 

Deviation 


Number 

Ranking 


Program completers demonstrate a good work ethic 
including punctuality, regular attendance, and 
doing what needs to be done. 


1.54 


0.88 


24 


Program completers can articulate career goals and 
plans for achieving them that are based on a 
realistic assessment of their interest, abilities, 
and the resources available to them. 


2.65 


1.19 


23 


Program completers manage their personal hves 
effectively, including budgeting, time 
management and accessing resources needed to 
overcome barriers to education and emplo 5 rment. 


2.74 


0.96 


23 


Program completers demonstrate the ability to work 
well with others. 


2.96 


0.86 


24 



The work with the focus groups thus yielded four potential indicators that 
could be useful in the evaluation of youth programs. Among an admitted 
limited sample of less than one-fourth of the total participants in the focus 
groups, the indicator related to work ethic was the most preferred By ranking 
the other three potential indicators, 23 of the 24 who returned completed 
forms, endorsed them as appropriate for use in evaluation. 
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SURVEY AND INTERVIEWS WITH PROVIDERS IN CENTRAL OHIO 



The four indicators developed from the focus groups became the basis of a 
survey conducted with providers of youth programs from the Central Ohio 
area. This survey was designed to determine how many providers collected 
information relative to the four indicators, and if they did not, the reasons why, 
and preferences for how such data should be collected. Providers were 
identified from three sources: a database developed for United Way and the 
Greater Columbus Chamber of Commerce by Opinion Strategies, Inc. and lists 
obtained from Job and Family Services of Franldin County and the Private 
Industry Council of Columbus and Franklin County. These lists were 
compared to eliminate duplications and 47 providers were identified. 

A one-page questionnaire presenting the four indicators was developed and 
faxed or mailed to the 47 providers that had been identified. The questionnaire 
is presented on the following page as Exhibit 6. The purpose of the survey was 
explained in the cover letter as follows: 

The Youth Employment Training Assessment is an initiative of The 
Center on Education and Training for Employment (CEH'E), a unit of the 
College of Education at The Ohio State University. As a project within 
the State University Education Deans Systems Integration Coalition, 
CEH'E was asked to study performance indicators and standards for 
providers of youth programs funded by the Workforce Investment Act of 
1998, for the purpose of obtaining stakeholder input for consideration by 
Workforce Policy Boards at the state and local levels. 

Enclosed is a questionnaire listing four possible indicators for use in 
evaluating youth programs. These indicators were developed with broad 
statewide input. Focus groups were conducted with over 100 
representatives of youth programs in nine of the Ohio School-to-Work 
regions. 

Now we need your input on these proposed indicators. We need to find 
out if providers of youth programs currently collect data about these 
indicators and how they would prefer to have data collected. The 
indicators that emerge from this research will be recommended for 
consideration to those implementing the Workforce Investment Act, both 
statewide and locally. 

After two weeks, those who had not responded were contacted again by fax or 
telephone to encourage their completion of the form. Two weeks later, a second 
round of telephone follow-ups were made. Even with two rounds of follow-up, 
only nine completed questionnaires were returned. This number is too few for 
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Exhibit 6. Questionnaire Sent to Providers of 
Youth Programs in Central Ohio 



Proposed Indicators 

Please tell us if you currently collect data on the following indicators for your training program 
and your preference for collection. If you do not, please indicate your reason for not doing so. 

1 . Indicator. Program completers can articulate career goals and plans for achieving 
them that are based on a realistic assessment of their interests, abilities, and the 
resources available to them. 

a. Do you currently collect this information? o Yes o No o Don’t Know 

b. If NO, why not? 

o Never been required o Do not have resources to collect 

o Do not feel it is important o Not applicable to my program 

o Too difficult to collect o Other ^ 

c. Would you prefer to collect this data, or would you prefer to have it collected centrally? 

o Self collected o Centrally collected o No preference 

2. Indicator. Program completers demonstrate .the ability to work well with others 

a. Do you currently collect this information? o Yes o No o Don’t Know 

b. If NO, why not? 

o Never been required o Do not have resources to collect 

o Do not feel it is important o Not applicable to my program 

o Too difficult to collect o Other 

c. Would you prefer to collect this data, or would you prefer to have it collected centrally? 

o Self collected o Centrally collected o No preference 

3. Indicator. Program completers manage their personal lives effectively, including 
budgeting, time management and accessing resources needed to overcome 
barriers to education and employment. 

a. Do you currently collect this information? o Yes o No o Don’t Know 

b. If NO, why not? 

o Never been required o Do not have resources to collect 

o Do not feel it is important o Not applicable to my program 

o Too difficult to collect o Other 

c. Would you prefer to collect this data, or would you prefer to have it collected centrally? 

o Self-collected o Centrally collected o No preference 

4. Indicator: Program completers demonstrate a good work ethic including 
punctuality, regular attendance, and doing what needs to be done. 

a. Do you currently collect this information? o Yes o No o Don’t Know 

b. If NO, why not? 

o Never been required o Do not have resources to collect 

o Do not feel it is important o Not applicable to my program 

o Too difficult to collect o Other 

c. Would you prefer to collect this data, or would you prefer to have it collected centrally? 

o Self-collected o Centrally collected o No preference 
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the calculation of any statistics. The only generalization that seems justified is 
that program performance indicators were not high among the priorities of 
those to whom the questionnaires were sent. Even stating in the cover letter 
that the indicators emerging from the research would be recommended to the 
state and local Workforce Policy Boards did not generate much interest. 

Four of the seven organizations that returned questionnaires reported that they 
collected data on at least one of the four proposed indicators. Attempts were 
made to conduct personal interviews with representatives of these four 
organizations to determine just what they collected and how they used this 
information, but only two of the four organizations agreed to cooperate. Edited 
summaries of these two interviews follow. Initial drafts were prepared and 
submitted for review to the individuals who were interviewed. The summaries 
that follow incorporate all suggested clarifications that were received from 
those interviewed. 



Village to Child Program, Ohio Dominican College 
1216 Sunbury Road, Columbus, OH 43219-2099 

Summary of interview with Joann Davis, Program Director, November 30, 2000 

Program overview. The Village to Child program was developed to involve the 
College in its surrounding community. Ohio Dominican College has reached 
out to young people from the neighborhood around its campus in an attempt to 
build linkages between the community and campus. In 1994, Ohio Dominican 
conducted a study to determine what residents of the neighborhood thought of 
the college. The college held focus groups, conducted interviews, and met with 
civic organizations in the community. The conclusions of this research 
indicated that the neighborhood held the college in low esteem. Conimunity 
residents considered it a detached and closed entity, a mostly white island in 
an area where the residents are predominantly African-American and the high 
school drop out rate is high. 

The neighborhood that surrounds the college provides a dramatic contrast to 
the campus. This area contains two of the city's largest trailer p^ks, the 
largest portion of the city's Section 8 housing, a number of transient 
apartments that rent on a daily basis, and a lack of sewers and sidewalks in 
some areas. The college was, in essence, "a white city on the hill in a black 
neighborhood." Community residents felt the college took no active role in 
promoting education in the children of the community. The first programs 
implemented by Ohio Dominican were created to increase the college s 
involvement in the community. 

The first undertaking was a one -week initiative on-campus program for sixth 
graders. One hundred students, mainly recruited by Ms. Davis, took part. The 



Columbus Foundation, which provided some of the funding, wanted data from 
this sixth grade program and the results were not as high as had been 
anticipated. In response, the College slowly developed the Village to Child 
Program. The village is defined as the neighborhood within the local zip code, 
43219. Each year 100 to 125 young people, with approximately 20-25 entering 
each year at the earlier grade levels, participate in the core program. 

The program has always been academically based, with a large cultural 
component and an emphasis on leadership development through community 
improvement. The program has evolved to meet the unique needs of the 
children. Studies show that African-American children tend to be more 
influenced by their peers than white children. Keeping this in mind, the 
program uses a family groupings model, with the students serving as members 
of the same large family. 

The program uses a cluster approach in mentoring the students. The students 
are assigned to a primary mentor, but may receive mentoring from several 
others. Ms. Davis, who ultimately serves as the head mentor to all of the 
students, decides the mentor cluster to which a student is assigned. Mentors 
are chosen on the basis of the student's interests and intended goals. Some 
mentors and students have been together for five years. Because the 
relationship between student and mentors is well established, the mentors are 
constantly aware of student plans and progress. 

The work component of the program began when eight graders who were 
completing the original program did not want to leave. Because many of these 
children needed job related skills and work experience, the Columbus Private 
Industry Council (PIC) became involved. The students are exposed to a college 
environment and often hold jobs at the ODC campus during the school year, 
mostly in food service. There is a continuing commitment to the students. The 
program also strives to obtain jobs for the students in the neighborhood 

One of the barriers to employment faced by the students is a lack of reliable 
transportation. Many of their parents do not own cars. To address this issue, 
the program provides a pick-up and drop-off transportation. Because all 
students are expected to attend, failure of students to be at designated pick-up 
sites often prompts fellow students to seek out absentee students. The 
students learn to look out for each other and function as a team. They are 
encouraged not to let each other down. 

The students are encouraged to attend the alternative high schools in the 
Columbus area and given four-weeks of preparatory study before they begin 
attending. This component was developed because the first group had 
considerable trouble academically. Eight weeks of study are provided for PIC 
students. Students who will enter middle school receive four weeks of academic 
preparation. One week focuses on science, and the next three weeks feature 



alternating classes in math, reading, and language arts. The students learn to 
move from class to class, as they have to do in middle school. The sixth-grade 
program emphasizes the sixth grade proficiency test. There is also an art 
program of two week for all students. The college’s resources are used to 
expose the students to dance, art, and computer technology. 



Measures of performance. Except for the forms used for the PIC summer 
programs, no standardized evaluations are recorded, but an on-going file exists 
for each student participating in the program. Assessment begins in fifth grade 
when students are asked what their career goals are. They are then asked to 
examine who in their family will help support them in pursuit of these goals 
and what types of training and skills their desired job would require. The 
students awareness of how to achieve these goals is enhanced by attending 
career fairs and visiting colleges. Local employers, including a division of the 
Department of Defense located in Ohio, send representatives to talk to the 
students about career planning and achievement. Many of these 
representatives also serve as mentors to the students. The DOD agency 
employs a high proportion of men involved in fields requiring math and 
science. The hope is that interaction with these men has a strong impact 
especially on m^e students who lack positive role models. These mentors 
discuss values, planning, careers, and school success with the students. 

For the PIC summer programs, there are weekly evaluations on the students. 
The forms for these evaluations are provided by PIC, and modified by ODC for 
the purposes of the program. These evaluations are documented. Attendance 
and punctuality are documented. The Columbus Public Schools issue 
students a "career passport" so many of their career goals are on file with the 
school system. The students also are eligible to receive an ODC identification 
card and password which allows them to use the computer facilities at the 
college. 

Last year thirteen students completed the program, nine of these students 
started the program as seventh graders and none entered it later than the 
eighth grade. Of these thirteen, eight were the first in their families to graduate 
from high school. High school graduation is the standard the program strives 
for because it represents a dramatic improvement in many neighborhood 
families. College admission is a plus, not the primary go^ of the program. 
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Huckleberry House, 1421 Hamlet Street, Columbus, OH 43201 

Summaiy of interview with Sally Murphy, Head of Youth Outreach, and Sarah 
Narendorf, Head of Transitional Living Program, December 1 1 , 2000 



Program overview. Huckleberry House offers a wide variety of services 
including a youth crisis shelter for walk-ins or referrals. It has served the 
community for 30 years and its programs have responded to the changing 
needs of youth. The facility includes dormitories for sheltered youths and 
administrative offices. Huckleberry House is a no-reject shelter and will accept 
anyone who meets legally established criteria. It is capable of dealing with 
severe mental and behavioral issues and has professionals trained to deal with 
these situations available on a 24-hour basis. 

Huckleberry House offers transitional programs for young people. The goal is to 
enable youth to make a successful transition back to their homes. Because 
this is not a possibility for every young person. Huckleberry House offers a 
transitional living program. This program sets up the young person in an 
apartment. While living there, the participant is required to be employed or in 
school. The program also seeks to help them develop skills for job readiness. 
There is consistent monitoring of student employment. 

A student typically stays in the transitional living program for an average of 14 
to 15 months. The target maximum is 18 months, but occasionally the time is 
extended for compelling reasons. A time frame of one year, the length of the 
typical apartment lease', is considered the ideal. 

Huckleberry House also has a rapid response team that is dispatched in 
response to various crises throughout the community. This team is under 
contract to other agencies to serve youth in crisis, such as school counseling 
for suicides. Huckleberry House offers a full after-care program and serves 
both family and youth. The Community Service Assistance program involves 
pre-employment and job finding skills. It serves as a coach in assisting youth 
with developing job skills and employment. 

Huckleberry House provides youth outreach. The employees and volunteers go 
where the most at-risk young people are, including runaways and homeless 
youth. School guidance counselors often refer youth to Huckleberry House. 
Students may become involved when their parents request assistance. Many of 
these youth in these programs are not immediately ready to obtain 
employment, but an emplo 3 onent assessment is conducted for each incoming 
youth. This is not done for outreach, and the transitional living prograim 
requires separate documentation. 
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Huckleberry House primarily serves the age group of 12-17, which offers a 
window to develop work readiness and employment opportunity. There are 
often many barriers to address before dealing with employment issues. 
Prevention groups meet after school to discuss drug and alcohol abuse 
prevention. The special problems faced by African-American young women are 
addressed in their meetings. 

Huckleberry House assists an under-served population, older youth who are 
not attending school. The community has few appropriate services for people 
who are 18 years or older but not ready for an “adult” environment or shelter. 
The staff works with faith organizations, and other non-governmental 
organizations to search the community to find resources for these older youth. 

Huckleberry House received no money under the Job Training and Partnership 
Act, and at the time of the interview received no funding through the Workforce 
Investment Act. It is working with the Private Industry Council on a program 
that focuses on putting people on welfare back to work. The grant is funded by 
the Department of Labor. 



Measures of performance. Data collected on participants include information 
on mental health, educational participation, truancy, and employment. These 
data must be collected on each participant as a requirement of the transitional 
living program. There is continuous assessment on employability. Much of the 
emphasis is on school proficiency tests, mentoring, and conflict resolution. 

Every participant has an individual service plan with vocational and 
educational goals. Goal attainment is evaluated when young people leave the 
program. Evaluation of the extent of goal achievement is ranked on a scale of 
1-5. There is no standard form used, evaluations are created on an individual 
basis. These evaluations include the steps for action and the needs to reach 
the stated goals. 

There is information collected at intake and discharge. The ability to get along 
with others is not a measure of performance, unless it is specifically included 
in an individual's service plan. Because the barriers to employment, self- 
initiative, and authority issues are very individualized, there are few standard 
measures. 

One standard by which all students are evaluated is the Daniel Memorial 
Assessment. This evaluation measures independent living skills, budgeting, 
and time management. This assessment was developed by the Daniel Memorial 
Home for Children, a community center located in Jacksonville, Florida. (This 
institution is now known as daniel Contact information follows.) Students 
take this test on entry into the transitional program, and repeat the test every 
six month to determine improvement. Students use the short form of the test. 
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which consists of 90 questions. The test can be taken orally or on paper. Most 
students take the paper version. The longer form of the measure has 280 
questions and is not used by Huckleberry House. 

In measuring outcomes, there is a conflict between individualized measures 
and generalizing to overall program outcomes. Huckleberry House focuses on 
tangible outcomes, i.e., the percentage of goals achieved by individuals. 
Educational goals are considered the most significant. 

Once the individual data are assembled, they are summarized as program 
outcomes. The information indicates pre to post-test improvement. 
Huckleberry House just started collecting this information within the last year 
and has not yet developed a systematic method of using it for program 
improvement. 



Contact information for daniel: 

The Daniel Memorial Home for Children has operated for 116 years and publishes 
many different educational and life skills materials for children and adults. It has 
recently changed its name to Daniel, aU lower case letters. Its address is 

daniel 

4203 Southpoint Bouleveird 
Jacksonville, FL 32216 
1-904-296-1055 
Website: www.danielkids.org 

E-mail: info@danielkids.org 
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GUIDE TO USE OF DATA 



In the original scope of work for the examination of the WIA indicators, a 
provider workshop had been proposed. The objectives of this workshop were to 
review the program performance indicators mandated by WIA and provide an 
opportunity for participants to comment on the utility of the indicators that 
had been developed from the focus group data. The poor response to the 
survey led project staff to reconsider their plans for the workshop. Only nine of 
the 47 providers who received questionnaires returned a completed form, even 
though the form was only one page long and could be completed in less than 
five minutes. This implied that there would be very few providers interested in 
attending a workshop that would mean a commitment of at least half a day. 

The interviews with the two providers reported in the previous sections 5 delded 
further, albeit limited, evidence that administrators are not linking data to 
program improvement. In these two programs, the data that are collected are 
used to plan and assess services provided clients, but not to guide program 
improvement. 

Given the limited current interest in performance indicators, a decision was 
made that rather than attempt to conduct a workshop, at which attendance 
would be questionable, a guide on how to collect and use data would be 
prepared. The guide was written with program administrators as the intended 
audience. It attempts to raise awareness of how data can lead to better 
programs by providing information on the degree to which objectives are being 
achieved. The guide was written as a primer, assuming no prior knowledge of 
research methods or statistics. 

A draft of the guide was sent to the same youth service providers to whom the 
survey of current data collection had been sent. The purpose of the guide was 
explained in the cover letter in this way: 

For the past two years, our Center has been working with providers of 
youth programs to develop potential performance indicators for use with 
their programs. 

Please find attached a draft copy of "Using Data to Improve Programs for 
Youth: A Primer". This document explains how data can be collected and 
used as a tool for improving programs such as yours. The publication 
attempts to provide a starting point for program administrators. 

We would like to identify administrators who would be willing to review 
and provide feedback on the enclosed document. Input from potential 
users will help us determine if the publication provides useful 
information and guidance. 
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If you would be willing to comment on this document, please reply to this 
message indicating your interest. At a time convenient to you, we will 
contact you by telephone to obtain your comments. 

The guide was mailed the last week in May 2001. By June 22, one 
administrator had responded. A telephone follow up was made with a 
representative of the responding provider, Ms. Kathryn Meyers, Marketing and 
Development Director of the Salesian Boys’ and Girls’ Club of Columbus (Ohio). 
Ms. Meyers was very complimentary of the guide. She felt it was the most 
useful explanation of the performance indicators for WIA youth programs that 
she had encountered. She suggested that the discussion of identifying the 
purpose for data collection be expanded and emphasized and this change was 
made. The final draft of the guide is included as an appendix to this report. 



O 

ERIC 



25 



32 



Conclusion 



The four potential indicators of the performance of youth programs that 
emerged from the focus groups essentially supplement the first of the core 
indicators mandated by the Workforce Investment Act. The first mandated 
indicator is as follows: 

Attainment of basic skills as appropriate, work readiness and 

occupational skills (WlA Sec. 136(b)(2)(A)(ii)(l)) 

The four indicators from the focus groups represent further definition and 
examples of what work readiness involves: career planning, managing one’s 
personal life, punctuality, regular attendance, and working well with others. 
The four potential indicators could be used to develop measures of the degree 
to which participation in youth programs develops such skills and behavior. 

The data this study produced indicate that few youth service providers in the 
Columbus, Ohio area currently use data to guide program improvement. This 
statement is based more on the poor response to the survey rather than the 
actual results it produced. Less than 20 percent responded to a one-page 
questionnaire and only two were willing to be interviewed about their collection 
and use of data. Only one of 47 providers responded with any comments to a 
draft guide on the use of data for program improvement. It appears that much 
training and development will be needed to prepare providers of youth 
programs to use data for program planning and evaluation. 
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Introduction 



The purpose of this guide is to provide an informative summary on the importance and relevance 
of data for program managers and stakeholders of youth programs funded by the Workforce 
Investment Act. The guide describes how data can be used to improve program effectiveness, as 
well as guidelines for collecting and analyzing data. The guide is intended for those not familiar 
with data collection or statistics. 

The Workforce Investment Act (WIA) has raised the standards of accountability for youth 
tr ainin g providers. The Act establishes new indicators and standards at both the state and local 
level for providers, which make the accurate collection and analysis of data all the more crucial 
and vital to the success of a program serving youth. A brief overview of the main provision of 
the WIA follows: 

Core Indicators of Performance for Youth Training Providers: (WIA Sec. 136 2, A) 

V- 

For providers serving youth ages 19-21: 

1 . Eligible youth who enter post-secondary education, advanced training, or unsubsidized 
employment. 

2. Eligible youth who enter an activity listed in indicator 1 who received a credential by the 
end of the third quarter after exit. 

For providers serving youth ages 14-18: 

1 . Attainment of basic skills as appropriate, work readiness and occupational skills. 

2. Attainment of secondary school diplomas and their recognized equivalents. 

3. Placement and retention in post secondary education or advanced training, or placement 
and retention in military service, employment or qualified apprenticeship. 
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Customer Service Indicators: 



The customer satisfaction indicator of performance shall consist of customer satisfaction of 
employers and participants with services received from the workforce investment activities 
authorized. 

Customer (employer and participants) satisfaction may be measured by surveys conducted at the 
conclusion of participation in the workforce investment activities. The U.S. Department of 
Labor American recommends the Customer Satisfaction Index (ratings on 3 scales from 1 to 10) 
for this measure, but local boards may adopt different measures if they yield comparable data. 

Additional Indicators: 

The State may identify in the State plan additional indicators for workforce investment activities. 
Local boards may adopt indicators beyond those required by the state. 

Target Population Groups: 

Youth must be ages 14-21, low income, and meet at least one of these six specific barriers to 
employment: 

1. Deficient in basic literacy skills. 

2. A school dropout. 

3. Homeless, a runaway, or a foster child. 

4. Pregnant or a parent. 

5. An offender. 

6. An individual who requires additional assistance to complete an educational program, or 
to secure and hold employment. 

Five percent may be non-low income if they have specified barriers to school completion or 
employment. At least 30 percent of the funds must be spent on out-of-school youth. 

The major difference in youth programs between the WIA and previous legislation is the 
requirement for continuous, year-around service. In the past, much youth programming had been 
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delivered in the summer, often as a combination of work and study. WIA calls for continuing 
service from enrollment until successful completion or termination. All providers must have 
available for all participants the following types of services; 

• Tutoring, study skills, and instruction 

• Alternative secondary school services 

• Summer employment 

• Work experience 

• Occupational skill training 

• Leadership development 

• Supportive services 

• Adult mentoring 

• Follow-up services 

• Guidance and counseling 

All providers do not have to offer all these services, themselves, nor do all participants have to 
receive all 10. Providers must, however, have established methods through referrals and 
cooperative agreements to make all 10 services available to all participants who need them. 

Providing such an array of services on an ongoing basis greatly increases the need for accurate 
data to track the progress and outcomes of participants. This guide is designed as a non- 
technical introduction on how to collect and use data to assess and improve programs. 
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Creating Meaningful and Effective Goals and Outcomes 



The first step to making use of data is to formulate clear, relevant goals and outcomes for a 
program. Collected and analyzed information can then be used to indicate if the program 
achieved its defined purpose, stated objectives, and desired outcomes. 

According to Levesque and others (1998), successful goals and outcomes have the following 
characteristics: 

They are meaningful 
They are realistic 

They are complimentary to the overall vision of the program 
They contain clear priorities 

They involve the knowledge and support of all stakeholders 
They are measurable 

Goals should be clear 

Well-written outcome objectives serve as a road map for programs by providing a clearly defined 
path for staff and participants to follow. Poorly written objectives lead to confiision and 
misplaced efforts (Ma, 1995). Objectives can be determined by questioning what outcomes are 
desirable, feasible, and realistic. Well-defined goals are more easily understood and more readily 
supported by all stakeholders. Goals should be written in terms of succinct, declarative 
statements beginning with "students will . . .", or "participants will . . ." (Wagner, 1995). For 
instance, a goal may state: "Participants will attain non-subsidized employment within six 
months after completing the training program". 

Goals should be specific 

Creating goals and outcomes is a difficult process because what is desired can be too broad or 
too specific to be realistic and relevant. Some goals simply cannot be quantified or measured, or 
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are so subjective that they cannot be a shared effort. For example, increasing student 
employment opportunities is an example of a goal that is simply too broad and subjective to be 
measured. Seeking an increase in student satisfaction with certain aspects of the program, such as 
employment placement services, however, is a more concise goal and can be measured through 
follow-up questionnaires and interviews. 

The Scope and Number of Goals should be manageable 

The scope of the goal and the number of goals can diffuse and frustrate efforts to measure 
progress by spreading resources too thin. It is important to focus on the needs judged to be most 
important. 

Stakeholders should be able to understand the objectives involved 

Goals need to be clear and concise from the onset so that everyone involved in the collection and 
processing of the data are looking for the same factors. All people filter information through their 
own internal baggage of assumptions, prejudices, and attitudes, so some variation of 
interpretation can be anticipated no matter how succinctly a goal is stated. For example, one 
observer may see a loud classroom as disruptive while another may view it as enthusiastic. Some 
observers may feel attendance is more important that performance, while others may feel the 
opposite is true. Precisely identifying desired outcomes and goals reduces this subjectivity, but 
cannot eliminate it. 

Goals can reflect existing policies 

Goals do not need to be new. They can be the stated objectives of a program, existing goals, or 
adaptations of goals set by other organizations or schools. It may be possible to adapt the goals 
of other institutions and programs for another program, but the ultimate concern should be the 
relevance of the goal to the program being studied. Rates of GED attainment achieved by one 
program, for instance, may not be indicative of the rates that should be achieved by another 
school, especially when other factors, such as age of school withdrawal, are taken into 
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consideration. Goals should address the unique nature of programs if they are developed 
specifically for the program 

Goals may be long term 

Goals may require long-term strategies to implement. Unrealistic expectations of the timeframes 
required can diminish enthusiasm and the perception of improvement. Realistic expectations of 
the time necessary to achieve goals should be an important component in developing the goals. 
Typically, only those employed by a program will "stick around" for long-term efforts, while 
business partners, parents, participants, and students will be more interested in short-term results. 

Goals Require an Assessment of the Current Situation 

Effective goals are grounded in the current reality of a program's strengths and weaknesses. 

Goals that do not accurately account for the program's current situation are often unfeasible. 

All Stakeholders should be identified and involved 

When determining goals, all stakeholders must be identified and involved in the process of goal 
setting. By involving all the stakeholders, the potential for redundant efforts is reduced and the 
maximum effectiveness of measurement is increased. Many programs exist to address pervasive 
problems, and similar programs in the same field should be consulted before beginning any data 
use projects that require new and extensive data collection efforts. This not only avoids duplicate 
efforts, but aligns goal setting with the efforts of those in same community and field, which can 
foster a sense of shared mission (Wagner, 1997). Stakeholders not only contribute to achieving 
the goal, but assist in the collection and identification of the information necessary to determine 
if the objectives of the goal have been met. 
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Goals must be tangible enough to be measured 



To be effective, goals have to be tangible enough to be measured, meaningful to the program, 
and contain clear priorities. For instance, the goal of increasing student attendance is easily 
measurable using past and present attendance records, relevant to the success of the program in 
which the students are enrolled, and clearly related to the progress of students in the program. On 
the other hand, a board based, subjective goal like involving students in program improvement is 
too vague, intangible, and subject to multiple interpretations. 



Using an Existing Mission Statement to Determine Goals and Outcomes 

An existing mission statement can be the foundation for creating goals and outcomes. A mission 
statement is a list of commitments and principles that a program seeks to uphold and enforce. 
Reflecting on a program's mission statement can be very productive. 

Holcomb (1999) recommends the following exercise for examining an existing mission 
statement and determining how to measure it in terms of collecting data. Stakeholders list the 
major points of their mission statement under the heading "What We Say". Stakeholders then 
determine how to prove or plan to prove that these points are met in the adjoining "Evidence We 
Have" and the "Evidence We Need" columns. This exercise is helpful in identifying what 
sources of data currently exist and for creating goals and desirable outcomes. This exercise can 
also suggest ideas and innovations in data collection methods and practices. The table is shown 
with hypothetical content (Holcomb, 1999). 
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Translating a Mission Statement into Outcome Measures 



What We Say 


Evidence We Have 


Evidence We Need 


Development of Academic 
Skills Necessary to be a 
functioning, productive 
citizen 


Entrance Tests 
Exit Tests 
Progress Reports 
Program Completion 
GED Completion Rates 


Instructor Input 
Continued Education and 
Training Statistics 


Attainment of employment 


Job Placements 
Employer Reviews 


Follow up information on 
-Salary 

-Employment Retention 
-Job Satisfaction 


Successful Development of 
Life Skills 


Life Skills Assessment 
Room Inspection Reports 
Money Management 
Assessment 


Follow up information on: 
-Use of community 
resources 

-Effective economic 
decisions 


Drug Free Atmosphere 


Drug Test Results 
Drug Related Expulsions 
Arrest Records 


Follow up information on: 
-Additional arrests for drug 
related activity 
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About Data and Data Use 



What Are Data? 

The word data is defined as a collection of things known or assumed such as facts, figures, 
evidence, records, and statistics from which conclusions can be drawn (Ma, 1995). Data are 
commonly collected on events (birth, death), people (demographics, school grades), and 
processes (programs, organizations). Some of the more familiar forms of data are the records 
routinely maintained by local, state and federal authorities and agencies, schools, and 
organizations. Data alone do not supply meaning, but people can use data to develop meaning 
(McNamara, 1999). Data create a foundation upon which decisions are formulated and questions 
answered. Data can be used to buttress arguments, prove points, illustrate changes over the 
course of time and create comparisons (Ma, 1 995). 

Using Data to Benefit a Program 

Data can be the foundation and profile of a program's successes and failures. Like an artist's 
portfolio, a collection of data presents a program's range of skills, its best products, and evidence 
of progress and acquired learning. Such a collection provides a umque picture of the program as 
a whole (Holcomb, 1999). Using data to analyze what is and is not working can determine the 
ultimate success of a program. Through the study and collection of data, areas of poor 
performance can be weeded out before additional time, money, and energy are wasted on them. 
Resources can be better allocated to achieve success, sounder strategies can be implemented to 
save time and money, wasteful portions of the program can be overhauled and successful 
strategies can be strengthened. Productive activities can receive more resources while failing 
initiatives can be phased out or subjected to intense improvement efforts. The time dedicated to 
the collection, analysis and use of data is usually well spent in terms of program success and 
support (Holcomb, 1999). 

Many programs are subject to public demands for accountability, and data can provide evidence 
of a program's effectiveness. Information acquired from data can be used to demonstrate the vital 
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roles a program plays in a community and encourage public and private support for a program's 
efforts. Funding sources, such as federal and local governments, often require data as verification 
that the money allotted to a program is well spent and worth the investment. 

Hendricks, (1996) has listed the following benefits that data collection and analysis can yield for 
a program: 

Increased recognition of successful strategies already in use 

More efficient and economical operation 

Identification of staff and student needs 

Improvement and updating of program content and delivery 

Improved community perceptions and contact 

More accurate understanding of intended and unintended outcomes 

More effective staff participation and input in improvement 

Motivations for Data Use 

The collection and use of data are motivated by both intrinsic and extrinsic factors. External 
pressures and requirements for results are usually the most common motivational factors for 
programs. Programs are accountable to various parties, including funding sources, students and 
participants, and the communities they serve. Internally motivated data collection originates and 
operates within the foundation of the program itself, spurred by the individual interests of the 
program. Internally motivated, program-specific data collection is usually the most effective and 
embraced approach. This motivation involves a commitment to providing proof in a manner that 
is meaningful to the person collecting the data and the program, rather than complying with the 
external forces of mandates (Holcomb, 1999). 

For instance, employees of a program are likely to be more enthusiastic at the prospect of 
collecting data to improve the program in a direction they have determined to be beneficial rather 
than simply performing routine collection for the sole benefit of a third party, such as a 
government agency. This is why it is essential to motivate stakeholders about the importance of 
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data to program success from the start of collection and analysis efforts, even if the collection 
must be done for a third party as part of an accountability process. 

The Challenges of Data Use 

As valuable as data may be, there is often resistance to using data as an informational tool to 
achieve desired outcomes. Six common difficulties, identified by Holcomb (1999) are listed 
below. 

Lack of Training and Familiarity 

The first problem is the lack of training and familiarity with data collection methods and uses of 
data as measurement tools. The very concept of statistics makes many people uncomfortable, and 
those who have taken statistics courses are likely to recall only the difficulties involved, rather 
than the possibilities and relevance to their program. 

Lack of Time 

The lack of adequate time is almost always a factor. Only so much time can be allotted to 
activities, like data collection, that are not immediately relevant, especially when a program is 
understaffed or overwhelmed by other activities. 

Feast or Famine 

Many data collection and use efforts are unintentionally made difficult because the collection 
methods result in an unexpected surplus and unmanageable amount of information. The sources 
of data may seem scanty at first, but once a thorough search and collection are underway, 
stakeholders are shocked by the resulting amount of information. The perception of scarcity and 
the reality of plenty overwhelm many data use efforts. 

Fear of Evaluation 

The fear of evaluation also hinders data collection and use. Not all outcomes of data use will be 
favorable and this can promote an anxiety about negative applications and hidden agendas. For 
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instance, stakeholders like teachers may fear the loss of their jobs based on student evaluation 
data. 

Fear of Exposure 

Apprehension over exposure also impedes data collection and use efforts. Inadequacies will 
potentially be exposed, a situation that may make many stakeholders uncomfortable and resistant 
to data use efforts. For instance, a stakeholder may be concerned about failing in the data use 
effort, thus exposing an inadequacy. 

Individual vs. Systematic 

The use of data can also challenge an existing institutional structure. Individual stakeholders are, 
in effect, taking collective responsibility for the effectiveness of a program. Working 
collaboratively often violates individual approaches to program improvement and replaces it 
with systematic efforts. Stakeholders who highly value their independent efforts and techniques 
may resist a program-wide effort. 

Another hindrance to effective data collection is the disruption of regular program schedules 
(Hendricks, 1996). Disruptions of program routines imposed by observers or time taken to 
complete interviews or questionnaires can create resistance to data collection efforts. 

Johnson (1997) recommends that a theme of continuous improvement should be stressed in data 
collection rather than the image of a fact-finding mission with the potential for singling out 
specific staff members. Vague reasons for requesting data will create apprehension and possible 
resentment. 

Using Data That Have Already Been Collected 

The challenge of data use not only involves the actual collection of data, but determining the 
relevance and effective use of those data already in existence. Most programs routinely record 
data regarding their participants, such as attendance records, completion rates, and program 
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specific information. Most of these data are routinely filed and rarely assembled together in a 
structured way to serve as a source of in-depth information. 

Often, no one person has complete knowledge of all the collection methods, data storage, and 
current use of existing data. This can have a negative impact on the ultimate success of students 
and participants, who may slip through cracks due to the ineffective dissemination of available 
data. A student needing tutoring in reading, for instance, may not receive it because test scores 
are not easily accessed. Some schools and programs routinely collect more data than they can 
effectively use, effectively rendering themselves data rich and information poor. The challenge is 
to use available data wisely (Johnson, 1997). 

Often data are used only for individual assessment of participants, rather than gathered together 
as measures of program progress. Seemingly random data, if properly aggregated, can serve a 
valuable purpose in defining goals and providing measurement of progress as well as a 
foundation for additional research. 

Usage of existing data can begin with archival data such as attendance records, grades, referrals, 
retentions, and standardized test results. These data create a program or school specific baseline 
by which comparisons to similar institutions can be made. Such data can typically be sorted by 
grade level, race, age, and socioeconomic status to produce even more informative patterns 
(Johnson, 1997). 

Existing data are a source of information that can be used to evaluate a program without the 
added expense or effort of additional data collection. Before beginning the process of collecting 
data to measure program achievements, it is helpful to review the methods currently in place for 
collecting data and to examine those data that are already collected to avoid duplicating existing 
efforts. 
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Collecting Data 



Identifying Stakeholders 

Determining all stakeholders is critical for the success of any data collection or analysis effort. A 
stakeholder is any interested or invested party involved in the program, including participants, 
parents, instructors, counselors, administrators, support staff, community representatives, and 
business partners. Each brings a different perspective to the data collection process. 

Singling out certain stakeholders will often result in a lopsided, biased approach to setting goals, 
collecting and using data. For instance, involving only the administrative staff may result in 
studies and goals that are related to budgetary issues, rather than the program as a whole. 
Instructors may not view data collection in terms of overall results, but from the perspectives of 
their students and individual classes. Program staff will often be interested in the long-term 
nature of data collection and use, while other concerned parties may be interested in those goals 
particular to their circumstances or agendas. Identifying and including all stakeholders reduces 
the likelihood of duplicated efforts, mismanagement and poor communication. 

Identifying the Purpose 

Accurately identifying the purpose of data collection efforts is the best way to avoid confusion, 
indifference, and possible resentment from stakeholders. The most important step is explaining 
to those who will be involved the reasons why the undertaking is necessary. An explanation is 
especially needed with those outside the agency, such as parents of enrollees in a youth program. 
The explanation should cover what you want to accomplish or find out. All communications 
should emphasi 2 e how the data collection will help the program and its stakeholders achieve 
their goals. (Hendricks, 1 996). 
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personal interpretations of the researcher or those being studied to affect the results. Objective 
measures apply the same standards of measurement to all data in a uniform way. 



Non-Reactive Collection Methods. Standardized tests, attendance records, and student grades are 
routinely collected and not subjective. (There is, of course, a high degree of subjectivity in a 
teacher’s assignment of grades, but that is not the meaning in this context.) These data are often 
called "internal sources" and are usually comprised of existing records collected on a routine 
basis. There is no reaction on the part of the students because they are unaware that they are the 
subjects of a data collection effort. Research of records also has a low degree of personal 
subjectivity on the part of the researcher since most records are based on vvddely accepted 
symbolic interpretations of student behavior and achievement, such as letter grades. There is no 
physical contact between the evaluator and the subjects of the study, and consequently, there is 
no reactivity. 

Examples of Non-Reactive Methods: 

Enrollment Records 
Attendance Records 
Budget and Program Reports 
Completion and Placement Rates 
Standardized Tests 

Collection Methods with Reactivity. A more direct method of data collection, such as observing 
a class in progress, produces a reaction among those being observed because they are aware of 
the observer's presence. The observations of the class are therefore affected to some degree by 
the presence of an observer. The sight of an observer collecting and recording information in a 
classroom is bound to alter the dialogue and dynamics of that class. Teachers and students may 
react to being observed by engaging in activities they assume the observer vvdll appreciate or 
expect, even if it deviates from the normal classroom routine. All information collected by an 
observer is also subject to the individual perceptions and interpretations of the observer. 
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Examples of Reactive Methods: 

• In-class observations 

• Completion of questionnaires 

• Focus groups 

• Interviews 

• Evaluations of student work or projects 

Data collection methods that involve intense, individual contact with students produce a high 
level of reactivity.. Interviews and questionnaires introduce information that can alter responses. 
There is always some ambiguity in responses since students may express answers in their own 
terms and not directly address or answer the questions put to them. Many students asked about 
the quality of a program may digress and name their personal grievances or list unrelated items 
and suggestions instead of supplying a direct answer. A participant asked about the quality of a 
work experience program may criticize the difficulty or getting to the work site, rather than 
address issues that are more pertinent. 

There is a difficult distinction between prompting an interviewee to answer a question and 
influencing the respondent to supply the response desired, or expected, by the interviewer. Such 
research can be highly influenced by the researcher's own perceptions. Each interview involves 
processing considerable input to select "relevant" responses to record and report. In a group 
interview (focus group) situation, the discussion can be dominated or altered by those 
moderating the interview or by dominant, outspoken participants in the group. 



External Data 

Supplemental data can be sought from external sources in the immediate community, similar 
programs and organizations, and academic institutions. For instance, many general statistics on 
youth are collected by government agencies, youth oriented programs, and organizations. 
Additional sources of data can be in the form of census records, unemployment rates, high 
school dropout rates, and other regularly collected statistics. These statistics can be used for 
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comparative purposes or to supplement those collected by an individual program. They can 
usually be attained with a little searching and can save considerable time in data collection 
efforts. For example, a program may wish to compare the poverty levels or unemployment rates 
of its participants with both national and state averages to gain perspective on the population it 
serves. 

Examples of External Data: 

• Census reports and demographic statistics 

• State and federal records 

• Poverty rates and unemployment statistics 

The most effective research makes use of more than one method for gathering information. 
Ideally, an effort to collect and analyze data would include a combination of non-reactive and 
reactive measurement methodologies. The basic research concept of triangulation, the use of 
multiple indicators to confirm a finding, is helpful in this regard (Holcomb, 1 999). Just as ships 
relied on several sources (astronomy, charts, and instruments) to determine their actual location, 
the process of determining progress should be based on several sources of data in order to 
achieve the greatest accuracy. 



Methods of Original Data Collection 

Focus Group (Group Interview) 

This method of collecting information involves the participation of a small group (8-12 people) 
who focus on the discussion of a particular topic. The conversation is led by a moderator so that 
it remains on the subject in question. 

Polls, Surveys, and Interviews 
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Participants respond to a series of uniform questions. Fixed-response answers alternatives are 
usually provided for the participant so that the results may be easily and accurately calculated. 
Surveys may be conducted with self-administered forms that can be completed in a group setting 
or sent through the mail. For those with access to the Internet, computer-based surveys are 
becoming more common. There is a web site, www.zoomerang.com, which provides assistance 
in designing questions and analyzing data at qiiite low costs. One-on-one interviews, either in 
person or by telephone, offer much more flexibility than self-administered forms. It is possible 
to follow up on comments and explore answers in depth. These advantages come with an 
increase in the cost of data collection and involve higher levels of reactivity with the potential for 
bias. 

Observations 

Observations may be performed openly or covertly. Covert observations are more difficiilt to 
arrange, and may raise legal and ethical questions. Observations are always subject to reactivity, 
even if they are covert, because observers select the behaviors they notice and record. 
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Using Data to Measure Goals and Outcomes 



One measurement or source of information alone cannot determine the achievement of an 
outcome or a goal based on multiple factors. The most feasible goals are those that can be 
accurately measured using multiple sources of data collection. For instance, a measurement of an 
increase in student enrollment in a skill-training program does not necessarily demonstrate or 
indicate improvement in the program as a whole. It may simply reflect an increase in the 
population in that age range, resulting in a temporarily higher enrollment rate. Enrollment is 
essential to the success of the program, of course, but it is must be supplemented with other 
measurements such as attendance, completion statistics, and placement rates to determine if the 
program is succeeding or not. 

Multiple sources of data are essential for developing accurate measurements. For example, the 
goal of attaining high levels of successful job placement for all participants would require data 
sources such as placement data, employer surveys, and follow up program completers to 
determine job retention (Levesque, 1998). 

Measurements should be clearly defined so that they may be accurately reported and applied to 
goals. 

Data Spreads 

Data can be spread out, or distributed, to define the overall character of program performance. A 
spread of data can highlight the strengths and weakness of a program. It is important to simplify 
data through the use of simple statistics, rather than an overwhelming amount of information. 
Single statistics, however, cannot provide a comprehensive picture of a program, and should be 
combined with other statistics to identify trends and patterns. The use of tables for comparison 
and illustration, along with summarized statistics, is especially helpful in conveying the "whole 
story" (Levesque, 1998). 



0"*ing Data To Improve Programs for Youth: A Primer 

ERIC 




20 



As a Single Statistic 



55% of program participants have earned their GED 

This statistic alone cannot convey complete information, but neatly sums up the combined data 
on GED earners in the program. 

As a Comparative Table'. 



Highest Level of Education Completed 


Percent of Participants 


Junior High Grades 6-8 


5% 


Some High School 


10% 


GED 


55% 


High School Diploma 


20% 


Some Community College 


1% 


Vocational School 


9% 



Measuring the Spread 

Data can be measured and evaluated by examining the range of the data. This is done by 
determining the lowest and highest values. The average, or mean, from these numbers is usually 
determined. Examining a range of data can identify patterns and undesirable results. For 
instance, a range of data in which an unacceptably low level of program completion is identified 
can be addressed with aggressive efforts to those increase those numbers. 

The Data Range of the Data Spread Example of program participant educational achievement is 
1% to 55%, which are the lowest and highest values, respectively. 
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‘’Outliers 



Outliers are the extremely low or high values that can distort or skew the outcomes of averages 
(means). Outlying numbers often deserve special attention. Removing outliers from the 
calculation of program averages can reveal more typical results. Care must be taken, however, in 
how outliers are defined. In the Data Spread Example above, the 55% obtaining a GED is the 
most extreme figure, 35 percentage points higher than the next highest figure. It is not an outlier, 
however, because it is reflects the education attainment of over half the participants. 

Outliers are frequently encountered when follow-up studies are done of the employment 
experiences of program completers. It is not unusual to find the most frequent earnings to be in a 
range of $8.00 to $10.00 per hour. A few former participants, however, may be earning two to 
three times this amount. If these atypical earnings are included, they have a strong impact on the 
average earnings reported for all participants. 

Arranging Data: Quartiles and Deciles 

Placing data in a range does not distribute data evenly, it only indicates the highs and lows. In 
order to determine if the data are distributed evenly, bunched, or concentrated, the use of 
quartiles and deciles is recommended. These methods subdivide the data so that these patterns 
are easily detected. 

Quartiles divide data into four equal parts. These four parts are the lowest 25%, the lower middle 
25%, the upper middle 25%, and the top 25%: 



Top 25% on a measure of interest 
Upper Middle 25% 

Lower Middle 25% 

Lowest 25% 



Deciles divide a distribution into 10 groups, each with an equal number. 



Q "sing Data To Improve Programs for Youth: A Primer 



22 



It is important to underscore that these distributions are based on the measure of interest, such as 

\ 

reading scores or post-program earnings. The number in each of the groupings is determined by 
the total number for whom measures are a.vailable. The utility of quartiles and deciles increases 
as the number being analyzed increases. It is possible to create quartiles with just four measures, 
but little is gained. When the number being analyzed becomes larger, patterns emerge. 

Translating Data 

In order to have meaning, data and statistics must be translated into concrete, universally 
understood terms. It is essential that all stakeholders be able to understand the significance of the 
data. Numbers may fail to convey the meaning if they are not translated into easily understood 
terms. For instance, a typical school year has 180 days. Average attendance of 92% may seem 
good at first, but the meaning changes dramatically if it is presented as an average of 1 5 days of 
absenteeism per student. 
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Presenting Data 



The successful presentation of data is determined by two factors; the clarity of analysis and the 
clarity of communication (Levesque, 1998). Identifying the intended audience before preparing a 
presentation is crucial. 

The methods of presentation, format and tone will vary according to the intended audience. The 
use of specialized jargon will alienate an audience that is unfamiliar with the field being 
discussed. Research results should be written and presented with the reader in mind. Terms and 
acronyms used exclusively by a program will not translate to an outside audience without 
clarification and simplification. Presentations should include visual aids, such as graphs, to 
express findings rather than rely on statistics alone (Ma, 1995). 

Displaying Data and Research Results 

It is crucial that the intended audience is immediately made aware of several important factors 
during a presentation of data and research results (Levesque, 1998). 

Who or What is Represented 

The audience will need to know who or what is represented in a presentation in the most specific 
terms possible. If the data are about single mothers involved in a job skills class, be sure to 
indicate that detail rather than labeling them as "generic" participants. 

Units of Measurement Used 

The unit of measurement used to determine results is crucial in understanding the relevance of 
the results. Measurement standards used exclusively by a program will require an explicit 
explanation to outside parties. Be sure all numbers are fully defined. If an audience caimot 
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determine if a munber indicts a percentage, a dollar value, or other value, a presentation will only 
cause confusion. 

Size of the Denominator 

It is vital to provide information on the amount of individuals or entities (programs, 
organizations, etc.) upon which statistics are calculated. These munbers serve as a base for the 
research and are essential to an understanding of statistical relevance. For instance, if a table 
provides results on a life skills test taken by unemployed youth, it makes a great deal of 
difference if the results are based on one program or several and if the number who took the test 
was 10 or 100. 

Time Period Covered 

Different time periods incur different data. Tests are administered during different seasons or 
semesters, and each year reflects certain demographic trends. If a study examines data collected 
over a long time period, that time period should be specified. Future trends and patterns can be 
erroneously assumed if the time period is not factored into a study. 

The Data Source 

Data sources should be acknowledged and noted in any presentation. An audience will be better 
able to understand the nature of the data presented when sources are noted. An audience is more 
likely to perceive that data are reliable and valid if they understand the attributing source. 
Identifying the data source also gives the audience a better perception of that subjectivity and 
objectivity of those data. 
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Tables and Figures 



Visual presentations of research results are an appealing way of presenting otherwise boring and 
overwhelming amounts of numbers and data. Key findings and important patterns are best 
conveyed through the use of visual aids that are easy to read and comprehend. 

Tables 

Tables are often more effective in presenting large amounts of information than graphs and 
charts. Graphs can be overwhelming if they contain too much information. Tables have the 
benefit of being able to contain more information, which can be highlighted or bulleted to 
emphasize important points or trends. 



Example of a Table 



Survey Results: How Many Participants Are Satisfied With Their Training? 


Percentage of 


1998 


1999 


2000 


participants, by year 


% 


% 


% 


Very Satisfied 


41 


34 


38 


Satisfied 


19 


22 


22 


Somewhat Satisfied 


25 1 


30 


30 


Not Satisfied 


15% 


14% ^ 


10% 



Figures 

Figures include visual representations such as pie charts, line and bar graphs, scatter and 
distribution charts. These are universally successful representations for diverse audiences, 
because the human tendency is to be visually oriented. These visual representations are 
employed to give quick conveyance of major trends, patterns, and results to an audience. The 
wise use of these visual aids will greatly enhance accurate understanding by an audience. They 
should be simple, easy to read and interpret. 
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User-friendly, accurate and clear results are essential for successful presentations. Graphs should 
be clearly labeled and not overly elaborate or colored to the point of confusion. 

Each axis should be distinct and defined. 



Example of a Figure 



Percent of Funding 2000 




BUfeskills 
B Job Skills 
□ Child Care 
El Vocational 
B Substance Abuse 
B Mental Health 



Conclusion 

The collection, analysis and presentation of data are essential to developing knowledge of how a 
program serves its participants and stakeholders, meets its goals, and fulfils a role within the 
community. While the collection of data may seem daimting at first, it is the basis of all future 
effort to improve programs, determine strengths and weaknesses, and develop strategies for 
serving participants effectively. 
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Appendix 



Guide to Academic, Life Skills, and Occupational Skill Standards 

Academic Standards 

The Achieve Standards Database 

A searchable database of state and international academic standards organized by grade level, 

subject, state, keyword, and topic. 

http://www.achieve.org/achieve/achievestart.nsf/pages/abtclr 

Education World 

Lists academic standards according to subject and state. Subjects are subdivided by grade level. 

http://www.education-world.com/standards/ 

Ohio Academic Content Standards 

A listing of academic standards for the state of Ohio. 

http://www.ode.state.oh.us/ca/ci/acs/default.htm 

Life Skills 

Life Skills for Vocational Education 

Full text of important life skill standards for success in the workplace and life. The site includes 
curriculum and teaching strategies. 

http://www.workshopsinc.com/manual/TQC.htiTi l 



McRel 

Listing of Life Work Standards broken down by grade level and type. 

http://www.mcrel.org/compendium/Standard.asp?SubiectlD=24 
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Next Step Career Specialists 

Listing of the National Occupational Information Coordinating Committee Life Skills 
Competencies 

http://www.nextstepinc.com/nex506Q.asp 



Occupational Standards and Competencies 
CSI Net 

Standards for career and technical education developed with existing academic standards by a 
consortium of seven schools. 

http://www.csinetwork.org/ 



National Skills Standard Board 

Contains a searchable clearinghouse of information on skill sets for occupations, certifications, 
and vocational education. 

http://www.nssb.org/ 



0*Net 

Lists skill sets associated with a large listing of careers. Searchable database. 

http://online.onetcenter.org/iTiain.html 



SCANS (Secretary's Commission on Achieving Necessary Skills) 

The Department of Labor listing of necessary competencies for schools and vocational programs. 
Contains printable guides, skills lists, and documents. 

http://wdr.doleta.gov/SCANS/ 

Vocational Information Center 

Contains articles on standards, tutorials, links, and information on specific skill sets. These 
include life skills, career skills, and information technology skill sets. 

http://www.geocities.com/ResearchTriangle/Forum/79 1 3/page3.htm l 
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